, UK Patent Application „„GB ,,,,2411492 ,,3, A 



(43) Date of A Publication 



(21) Application No: 

(22) Date of Filing: 



0404143.0 
25.02.2004 



(71) Appllcant(s): 
Patsystems (UK) Limited 
(Incorporated in the United Kingdom) 
22 Stiand Street, LOIMDON, SE1 2ES, 
United Kingdom 

(72) lnventor{s): 
John IVIcGinley 
Ian Greaves 

(74) Agent and/or Address for Service: 
Alistair Hamilton 
Ty Eurgain, Cefn Eurgain Lane, 
Rhosesmor, MOLD, Flintshire, CH7 6PG, 



(51) INTCL^: 

G06F 17/60 // G06F 11/30 11/34 



WO 2003/107607 A1 



(56) Documents Cited: 
GB 2379063 A 
WO 2002/001472 A1 
US 5872976 A 



(58) Field of Search: 

UK CL (Edition W)G4A 
INTCL^GOeF 

Other: Online: WPi, EPODOC, PAJ. OPTICS 



(64) Abstract Title: IVIanaging quality of service in an electronic trading system 



(57) An electronic trading system for trading In e.g. financial instruments, comprises a quallty-of-service 
(QoS) subsystem, which is operative to impose limitations upon trading activities in order tiiat the 
performance of a component of the system or of the system as a whole is maintained within specified 
tolerances. For example, it may limit the number of events that can be initiated by a trader. It may also 
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such messages have a particular content (e.g. are inherently urgent in nature or essential to proper 
operation of the system) or are to or from a privileged user. The system may also have an integrated 
protocol stack for routing of data to enable the location of data bottlenecks to be identified. Messages 
may be processed according to a token bucket algorithm. 
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Electronic trading system 

5 This invention relates to an electronic trading system. It has a particular application to a 
system for trading in intangible things, financial instruments. However, it might also 
find application in trading in anything that mi^t be traded in an electronic market. 

Providing a trader with an effective interface to an electronic market provides a 
considerable technical challenge. To be effective, the trader must be presented with 
10 accurate and timely information relating to the state of the market, and the trader must 
be able to buy and sell within the market at a known price. The complexity of the entire 
system is considerably complicated by the fact that there are a variable number of 
traders active at any one time, and that it is not possible to predict accurately when any 
one of them may initiate a trading request. 

15 In addition to straightforward performance in processing of transactions, it is also of 
great importance that the performance is maintained within well-defined limits. That 
can be expressed as a guaranteed quality of service (QoS). 

The ultimate performance of the trading system may depend upon the hardware upon 
which its software is executing and upon fixed infi:astructure, such as 
20 telecommunication links, that caimot economically be upgraded to cope with the 
maximum anticipated system load. Therefore, an aim of this invention is to provide a 
trading system that can offer a required QoS to a trader interacting with an electronic 
market. 

From a first aspect, this invention provides a trading system comprising a quality-of- 
25 service (QoS) subsystem, which subsystem is operative to impose limitations upon 
trading activities in order that the performance of the system as a whole is maintained 
within specified tolerances. 



2 



Very generally, the QoS subsystem imposes limitations upon specific activities to 
preserve the overall well-being of the system. It ensures that users are not permitted to 
make demands upon the system that go beyond the capacity of the platforms on which it 
is implemented. 

5 As a first example of its function, the QoS subsystem may impose a limit upon the rate 
at which data can enter the system. For example, it may limit the number of requests 
that will be accepted on an input. For example, it may control the number of requests 
that can be made in a time slice. Within that time slice a limit may alternatively or 
additionally be placed on the size of burst data that may be received into the system. 

10 Suitably, the token bucket algorithm may be used in order to limit the flow of requests 
into the system. This algorithm is commonly used in computer networking to control 
the flow of data packets in a network and can limit throughput in a moving timeslice 
rather than in fixed, periodic time slots. However, the advantages that it provides are 
not generally recognised by those skilled in the technology of this invention. 

15 Where operating regulations allow, it may be advantageous to provide a level of service 
that is dependent upon the identity of a user from which a service originates or to whom 
it is directed. Thus, the system may, at any time, allow a request to enter the system 
conditionally upon the source or destination of the request. It may also be dependent 
upon the nature of the service. 

20 An important aspect to the control of QoS is control of all aspects of data transport 
within the system. Therefore, it is particularly advantageous that a single integrated 
metric stack handles all data transportation within the system from the business level 
down to the hardware level. 

A further preferred feature of the QoS subsystem is an ability for the system to measure 
25 its performance and dynamically reconfigure itself based on these measurements to 
ensure a defined level of quality-of-service. For example, the system may provide the 
ability to intelligently shed load based on business priority, intelligently delay updates, 
operate in a distributed manner (requiring no centralised point of control) and limit 
bandwidth consumption to a predefined maximum at a business defined application 
30 level. This is in contrast to the simpler concept of limiting load at a network level. 



A trading system embodying the invention may incoq)orate priority-based routing. 
That is to say, the QoS subsystem may be operative to assign a priority to a message, 
messages with a high priority being handled in preference to those with a low priority. 
The priority may be determined in accordance with one or more of the sender of the 
message, the recipient of the message or the content of the message. For example, the 
priority may be a numerical value that is calculated by addition of contributed values 
derived from one or more of the sender of the message, the recipient of the message or 
the content of the message. 

The QoS subsystem may be operative to control latency and accuracy of 
communication of data from the trading system to external client applications. For 
instance, the client application may request that the data is sent as fast as possible or that 
data batching may be applied. In effect, a client can connect and request that the system 
batch data (high latency) but that all changes must be sent, or the client could request 
fliat a low-latency link be established and that only flie latest data is required. 
Moreover, the client application may request that all data changes during a period are to 
be reported or that only the latest data be reported. 

Conveniently, the QoS subsystem may monitor performance of the application by way 
of Java management extensions. 

More generally, a trading system embodying the invention may use a rule-based system 
to control alarm reporting, fault diagnosis and reconfiguration. This provides for a great 
amount of flexibility in configuration of the system. 

An embodiment of the invention will now be described in detail, by way of example, 
and with reference to the accompanj^g drawings, in which: 

Figure 1 is a diagram showing the principal logical layout of a system embodying the 

invention; 

Figure 2 is a diagram showing the object broadcast bus, being a component of the 
embodiment of Figure 1, and its link to the QoS subsystem; 
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Figure 3 is a diagram that illustrates the design of this QoS module of the embodiment 
of Figure 1; 

Figure 4 is a diagram that illustrates interactions between the MBeans and bean pools in 
the QoS subsystem; 

5 Figure 5 illustrates various parameters measured by the QoS subsjretem in the 
embodiment of Figure 1; 

Figure 6 illustrates monitoring of response time of objects within the embodiment; 
Figure 7 illustrates the operation of request bandwidth control; and 
Figure 8 is a diagram illustrating operation of the "token bucket" algorithm. 

10 The invention will be described in the context of an electronic trading platform. The 
overall system is based on the 'layer' pattern. The following diagram presents the high- 
level logical view of the system. Note not all packages are displayed; only those that 
show significant architectural concepts. Moreover, many of the packages are not of 
direct relevance to the invention and are described only to the extent required to place 

1 5 the description of the invention in context. 

This embodiment is implemented using the Java language, and it is assumed that the 
skilled person to whom this description is addressed is familiar with Java and associated 
technologies. However, it will be understood that a Java implementation is merely a 
preference and is not essential to the invention. 

20 The Layers 

The following sections detail the role of the components within the system, and the 
interaction between the layers of the system. 

Infrastructure Layer 

The infrastructure layer provides the basic functionality required for the system such as; 
25 persistent data storage, a standard interface for access to asynchronous messaging, a 
system wide accessible mechanism for event logging, a system wide mechanism for rule 
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processing, a centralized system for security and access control and a system wide 
service location facility. 

Domain Layer 

The domain layer provides a set of shared resources for executing the business 
processes of the system such as order entry, price distribution, contract (instrument) 
management and message routing. This layer should be thought of as providing a set of 
'services' that can be consumed by the application layer. In this respect the architecture 
is similar to the 'service oriented architecture' employed in the web services field. The 
following diagram shows how interfaces are exposed fi-om the domain logic layer and 
aggregated by the application layer to provide different applications via the use of a 
'virtual' service bus. 

Application Interface Layer 

The application interface layer acts as an aggregation of services provided by the 
domain layer and provides the distribution protocol for inter/intra-net connectivity. The 
packages in this layer aggregate services provided by the domain layer into the 
applications that are required. 

Presentation Layer 

The presentation layer handles how tiie screen rendering is conducted. It contains the 
minimum logic required to achieve this goal. It contains a screen rendering package, a 
lightweight object proxy implementation and a communications library package. 

The Packages 

This section provides a brief overview of the responsibilities of each of the packages 
within the system. This is only intended to give a brief overview of what a package 
does and is not a comprehensive description of the responsibilities of each package. 

Swing 

This package is concerned with providing the graphical components required for screen 
rendering for the entire system. It is based on the Java Swing classes. 
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Object Proxy 

This package is a very thin object proxy implementation simply to support the client 
side access to the concrete objects within the application interface layer. 

Communications Package 

5 This package contains the code required for intra/inter net communications. This 
package is deployed both in the application layer and the presentation layer. It supports 
the use of TCP/IP (via SSL/TLS), serialized objects over HTTP(S) and XML over 
HTTP(S). 

Trading Client (TC) 

10 The TC is responsible for aggregating the functionality required for a user mteractive 
trading application and providing the statefull session management of this connection. 
The services for submitting, amending, cancelling orders and receiving prices are 
aggregated together to provide the trading application. 

Systems Administration Client (SAC) 

15 The SAC is used to configure the standing data in the system such as contracts, user 
accounts, order routes etc. The services such as contract configumtion, editing user 
accounts and setting passwords are aggregated to provide the system administration 
application. 

Risk Administration Client (RAC) 

20 The RAC application provides the pre-trade risk permissioning and the post-trade risk 
monitoring within the system. The services for editing account limits, monitoring risk 
parameters and editing risk system rales are aggregated to provide the risk management 
system. 
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Financial Information Exchange (FIX) interface 

The FIX interface package provides a non-interactive (non GUI) route into the trading 
system and is primarily designed to service FIX message 'pipes'. It aggregates services 
such as order submission, amendment and cancellation. 

5 Fill interfece (FIL) 

The FIL interface is another example of non-interactive connections with the system 
and is supplied to provide a feed of fills out of the system for use by third party back 
office systems such as Ralph & Nolan. It aggregates services such as fills. 

System Monitoring Client (SMC) 

10 The SMC's primary role is to provide an electronic 'flight-deck' view of the system 
components and reporting system performance and fauhs. Its primary user would be 
technical support. It aggregates the services provided by the Quality-Of-Service (QOS) 
package and the statistic services provided by the other domain packages, such as 
message throughput, idle time, peak load etc. 

1 5 Object Broadcast Service (OBS) 

The OBS handles differing requirements for broadcasting updates of objects (i.e. orders, 
prices) to a client application. 

The first is to broadcast an update (object alteration) to many specific clients, ignoring 
other logged in cHents, such as a change to an order, which should go to every logged in 
20 trader in that broadcast group, even if they didn't implicitly request notification for that 
object. 

The second requirement is to broadcast an update (object alteration) to many clients, 
this time not using a broadcast group but based on the objects the client requested. For 
example, a price update must go to many clients but only the clients that requested this 
25 price (object) and the clients may be in differing broadcast groups. 



The OBS is a pool of stateless beans that store these object to application mappings, in 
effect an application subscribes to an object. When the OBS is infonned of an object 
update, it broadcasts the change to all subscribed applications. 

Risk Management System (RMS) 

The role of the RMS package is to provide both the pre-trade risk management (onier 
permissioning) and post trade risk monitoring (profit & loss). It provides services that 
are accessible primarily from the RAC but could also provide services such as profit & 
loss services to be consumed by the trading application if required. 

Order Management System (OMS) 

The role of OMS package is to provide the services required for placing, amending, 
cancelling and querying of orders. In addition to providing these services the OMS also 
takes care of the in system execution of orders (see the Managing Orders Use-Case) 
where necessary. It manages orders from many users so is in effect a shared resource, 
and can be run in parallel. 

The OMS can be parallelised because in the majority of cases orders are independent 
from each other. For example, a trader places a limit order then places a market order, 
these two orders are independent in how they are run, in other words there is no 
interaction between these two orders as far as reporting order states, processing fills etc. 
is concerned. Because, orders are independent there is no requirement to have all orders 
for a user or TAG registered in the same OMS. An exception to this rule is where a 
multiple leg order (for example and OCO or MEL) is entered and in this case all legs of 
the order must be registered and managed from the same OMS. 

The OMS also has the role of managing the triggering of synthetic onlers such as the 
Stop and Market-If-Touched. 

Order Book Management System (OBMS) 

The OBMS provides services such as order status notification, order status querying, 
order fill processing, and the provision of segmented views based on individual 
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user/account details of the centralized order book. It also provides a 'centralised' 
account position service. 

Applications such as the trading client and risk administration client register interest in 
receiving information and updates from the OEMS, which responds to input events 
5 from the OMSs and fill interfaces. The rationale for dividing order information from 
the actual order is that some client applications may need to access order information, 
for example history, current status and fills, but may not be allowed to directly affect the 
order, for example cancel or amend it. Equally there may be the requirement to allow 
orders not entered via the system to be represented in the system, for example 
10 processing fills and orders entered via a different trading system. In this latter case, 
there is no concept of the order within our system and it can therefore not exist in the 
OMS, but we must be able to display the order and maintain its position. 

Contract Management System (CMS) 

The CMS provides services to locate and download information describing tradable 
15 entities. It provides the interfaces to obtain execution point and instrument-specific 
(commodity and contract) information. 

Price Subscription Controller (PSC) 

The PSC provides a centralized point for access to and a subscription/mechanism for 
application layer packages to access price information using batching and polling 
20 methods. Note the components within the Domain Layer (and certain high performance 
application layer applications) directly access price information of the 'PriceBus' and 
do not obtain price information from the PSC. 

Administration S}^tem (AS) 

The AS provides the services required for administering the system. For example 
25 allowing contracts and user accounts to be configured, order routes to be configured etc. 



10 



Data Store (DS) 

The DS is responsible for serving the domain and application packages with the data 
objects within the system such as orders, contract configuration, user accounts, trader 
accounts etc. It provides a global repository for read and write operations on objects, 
caching of the objects stored via the persistence package of the infra-structure layer, 
operates in a lazy read mode, and automatically manages stale data. 

All read and write operations on data objects, that must be persisted, go via the 

DataStore. 

Message Routing System (MRS) 

The MRS supports the routing of messages between domain layer packages, based on a 
database of message routing rules. It operates in a stateless manner and coordinates the 
consumption and delivery of messages to and from the queues, which link the domain 
packages together. The MRS initially uses MOM queues to communicate between 
system components but should be treated as a fa9ade allowing a different 
communication system (TCP/IP, e-mail) to be used as appropriate. 

ESA Adapter (ESAA) 

The ESA (ESA Adapter) acts as a 'bridge' between the EE system and the legacy ESAs. 
It contains four interfaces these being the orders, fills, prices and configuration data. 
Additional interfaces may be designed dependant upon specific exchange requuements. 

Exchange Gateway (EG) 

The EG implements the interface to the exchange-specific gateways. They implement 
four interfaces, these being a prices interface, an orders interface, a fills interface and a 
standing/configuration data interface. The internal workings of the EGs are specific to 
each exchange. 

Quality of Service (QoS) 

The QoS is responsible for monitoring and gathering the various QoS parameters 
required fiiom the system. It also provides these parameters via a set of services to the 
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SMC. In addition to this it can be configured to apply a set of rules and if warning or 
errors are detected and log these via the Log4J package and also if required initiate 
alerts to administration staff. 

Security and License Provider (SLP) 

5 The SLP manages the security logon requests and authentication of users and modules 
within the system. 

Persistence Facade (PF) 

The persistence fa9ade provides a coherent interface for persistent storage within the 
system. It provides storage via JDBC to a third-party RDMS vendor and to disk. 

10 Communications Fa9ade (CF) 

The communications fa9ade provides a coherent inter&ce for message queuing and 
publish-subscribe via JMS to a third party MOM vendor. 

Rule Engine (RE) 

A third-party rule execution engine is employed within the architecture to provide the 
15 user-defined order routing requirements of the MRS. In addition, the rules engine can 
be employed by the RMS, if required, to provide more complex rule-based order 
permissioning. 

Logging Package (LP) 

A third party logging API is used within the system to provide the ability to; 
20 • Log messages in a system wide consistent manner. 

• Support varying formats of log file for example plain text, HTML, XML or 
binary format messages 

• Persist messages via JDBC 



• Send log entries via JavaMail (SMTP,P0P3,IMAP, etc) 
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• Manage log entries via JNDI. 

This package may require extending to support transactional logging. By utilizing fee 
same logging method across all packages we provide a consistent system wide format of 
logs. 

5 Service Locator (SL) 

This package provides centralized and abstracted access to JNDI services. Multiple 
clients use the service locator, thus reducing complexity and improving performance by 
caching previously identified resources. 

Summary 

10 This section has shown how the architecture of this embodiment is divided into distinct 
logical layers, from the basic system wide functionality in the infiBstructure layer 
through the business logic of the domain layer, to the aggregation of fliese business 
services in the application layer Ihen onto the presentation layer. Cross-cutting 
concerns such as logging, auditing and security have been addressed by providing 

15 centralised fiinctionality in the infrastructure layer in the logging package and the 
security and license provider. Vendor dependencies on RDBMS and MOM have been 
abstracted and placed in specific components within the system, in the persistence 
fafade and the messaging fa9ade components respectively, therefore reducing the 
rework required to use other third party applications. 

20 Vendor dependencies due to application server (AS), which generally (although not 
exclusively) amount to JNDI lookups, have been isolated into the Service Locator 
package. This Service Locator also acts as a caching interface to JNDI to improve 
performance. 

The responsibility for message flow through the system is decoupled fi-om the 
25 components to a discrete messaging subsystem that uses user-defined rules to govem 
message flow. This provides flexibility in how components can be deployed and the 
interactions between components. 
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By providing a broadcast concept into the distribution of prices the embodiment 
delivers efficient price distribution, both in terms of speed and bandwidth usage. A 
price concentrator/repeater pair and a price distribution service are capable of batching 
price updates and delivering them via XML over HTTP. Although multicast does not 
supply reliable delivery of packets, with the application of the JavaGroups software the 
system can build up a sequenced and reliable protocol stack if required with no 
architectural impact. 

Having described the context within which the various aspects invention can be 
implemented in a manner that will be clear to those familiar with the technical field, the 
specific parts of the system that provide the functionality will now be described in more 
detail. 

Specific objects in more detail 

The QoS module and those subsystems and modules witii which it interacts will now be 
described in more detail. 

The Object Broadcast Service (OBS) is a subsystem that asynchronously sends object 
updates to the relevant client instances. It is described here because the return route 
from the domain layer to the application layer for many of the objects (orders, fills, 
prices, contracts) is llirough the OBS and its proper operation is therefore critical to the 
level of service that the system can provide. 

Figure 2 illustrates the main components of the OBS. The OBS is based upon 
JavaGroups, which is a technology that implements reliable multicast communications 
between group members based on IP multicast and a configurable protocol stack. The 
function of JavaGroups will be appreciated by those skilled in the technical field, and 
this will, therefore, not be described in detail here. Further information, should it be 
wanted, can be found in JavaGroups User's Guide, Bela Ban, Dept. of Computer 
Science, Cornell University. 

All object updates are notified to an object broadcaster that runs as a Java process 
outside the application server. The object broadcaster broadcasts onto a JavaGroups 
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channel. Every communications stub (to be described) receives these object updates 
and filters tbem to ensure that a client application only receives the relevant updates. 

As a client connects to the system, a communications stub is first created on the 
application layer. This communications stub is assigned a broadcast group based on 
5 information relevant to the application and user that connected. This information is 
retrieved as part of the security checks carried out by the security and license manager. 
The communications stub then creates a JavaGroups channel and connects onto the 
object broadcast bus. 

Whenever an object is updated, the relevant domain component (OMS, RMS, IMS etc) 
10 issues an RMI call to its relevant object broadcaster. The object broadcaster simply 
broadcasts this object update onto the object broadcast bus. Every communications stub 
within the application layer will receive the object update. Each stub then filters the 
object based upon its own internal filter chain to ascertain if it can forward this object 
update. If a stub is required to forward this update, it then issues an update object call 
15 to the communications protocol converter and thence to the client application in the 
presentation layer. If the object is not to be forwarded the stub simply ignores the 
object update. 

The QoS component (to be described in more detail below) listens to the object 
broadcast bus and gathers statistics on the number of object broadcasts being generated. 
20 It also monitors the broadcast group for communications stubs joining and leaving the 
bus. Additionally, it monitors for component failxires, which is supported by deploying 
the group membership component within the JavaGroup protocol stack. 

The QoS Subsystem 

Quality of service monitoring is an integral part of the trading platform. The role of 
25 QoS is to monitor the system resource utilization, allow dynamic reconfiguration of 
components, allow dynamic fault investigation and provide a feed of data in an 
industry-standard form that can potentially be plugged into existing management 
consoles. To this end, the use of Java Management Extensions (JMX) has been adopted 
into the trading system architecture. The QoS management within this architecture is 
30 focussed at the business process and application level, rather than at the lower 
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networking level. Software infiastracture and hardware infiastructure management can 
be embedded into the system through use of third party MBeans if available. 

A standard logging package Log4J, managed by the Apache Software Foundation, 
provides a system-wide standard for logging, extended to support transactional logging. 
5 For example, the system can start a transaction to log messages, errors and other events. 
It then can either commit the changes, whereupon they will be forwarded to the log 
sink, or rollback the log, effectively throwing away all entries logged within the 
transaction. During a transaction, logging is not committed to disc to improve 
performance: only upon commit is the log flushed to disk. Auditing works in a similar 
1 0 manner however does not support the transaction control. 

Figure 3 shows the overall design of this component and how it integrates into the rest 
of the system. 

The major point to note is that an MBean (a Java object that represents a manageable 
resource, such as an application, a service, a component, or a device) is deployed on a 
15 per-pool basis to allow tiie monitoring and management of the entire bean pool. 
MBeans can also be integrated through a standard MBean server to allow the 
monitoring and management of applications and of the software infiastructure as well, if 
required. 

The interactions between MBeans and pools will now be described. Figure 4 shows 
20 how upon creation of a pool bean by invoking the method ejbCreateO, the relevant 
MBean is located by the ServiceLocator. The pool bean (most typically a stateless 
session bean) then registers itself with its manager bean (MBean). The MBean updates 
it internal statistics, for example, how many beans are currently in the pool, rate of 
creation/destruction etc. Then the instance of the bean (EJBObject) is stored in the local 
25 cache of the MBean. The MBean then issues an update signal via the MBean Server so 
to inform any QoS user interface of the latest state of the pool. 

As the QoS user interface issues management functions, these are relayed via the 
MBean Server to the relevant MBean. The MBean then issues multiple method calls to 
all of the beans within a pool by referencing its internal cache of EJBObjects. Likewise, 
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as each bean issues a notification by invoking the update(...) method, the MBean 
processes these muhiple calls and then makes an update (...) method call containing the 
relevant data as required. 

When the contamer removes the bean fi:om the pool using the method ejbRemoveC ), the 
5 bean must call the deRegiSter( . . . ) method to inform the MBean to remove its 
reference from its local store and also issue a new update (...) message to the MBean 
server. 

The above describes the basic architecture of the manner in which JMX is enabled 
wilhin the system. Attention now turns to the method of alann generation, alarm 
1 0 management and remote notification. 

Within the MBean specification are specific beans that implement counter and gauge 
functionality. These are initially employed to produce the required trigger events. 
Timer beans are used to trigger statistical update events based on a predefined time 
period. The QoS Management application is configured to receive these notifications 
15 and to act as a central repository of messages. These events are transported to the QoS 
management application through an RMI connector, which itself is implemented as an 
MBean, allowing it to be dynamically loadedAmloaded as required. 

The QoS manager can also access the rules engine (if required) through the rule engine 
bean. This allowing the implementation of specific customer rules with no change to 
20 the application. The JavaMail API is used to support SMTP and POP email 
communication. This allows the management application to issue alerts and reports to 
maintenance personnel, who may be remote bom the site at which the system is 
installed. 

In more advanced embodiments of the invention, the QoS manager may be extended to 
25 actively manage the system. For example, the QoS manager may change bean 
configuration parameters, and alter application server or message queuing parameters 
during while the sj^tem is running. 

Centralised logging is also integrated into the system through the use of Log4J and 
using the JMX support that Log4J provides. This allows the system to alter logging 
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levels and parameters dynamically during run-time. It also supports the automatic 
notification of alarm conditions directly to the QoS manager without the need to scan 
log files on disc. The actual method of logging to disc is by the Log4J SocketAppender 
and SimpleSocketServer. This allows multiple writers to asynchronously write to the 
5 same log file. By decoupling the write/store process through a network connection, the 
actual process of writing to disc may be offloaded onto another machine. This approach 
may also be used for producing the audit file. 

The parameters that the QoS subsystem monitors and logs will now be described. 

The QoS subsystem can be considered as operating at several levels within the system. 
10 In this embodiment, the levels are defined as follows: 

• level 1 - hardware and infi^tructure monitoring; 

• level 2 - software infi-astructure monitoring; 

• level 3 - appHcation monitoring; and 

• level 4 - business process monitoiing. 

1 5 Monitoring at the lowest level, level 1, enables hardware fiiults to be identified and load 
to be measured. Level 2 monitoring enables faults in software infi:astructure 
components, such as databases and message oriented middleware faults to be identified. 
It also allows load within the software infitistructure to be measured. At level 3, 
monitoring enables end-to-end monitoring of an application. This monitoring is 

20 business process agnostic and provides measures on how well an application is 
performing regardless of its business use. The highest level, Level 4, is concerned with 
monitoring how well a business process is fimctioning. 

For example, assume that users experience order processing (a business process) is 
running slowly. Without the ability to drill down to the application layer, this 
25 information is of little use. However, if monitoring at level 3 reveals that the order 
management system is performing slowly, a technician can fiirther drill down through 
layer 2 to discover, for example, that the database writes-per-second rate is low. This 
might lead on to investigation at level 1 which might, for example, reveal that the discs 
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are full. Although this is a trivial example it demonstrates the need to be able to 
navigate down through the layers of a system. Monitoring at level 1 is a common and 
well-understood system implementation task and will not, therefore, be described 
further. The QoS management component is designed to address the needs of levels 2, 
5 3 and 4. 

Equally, active management of the QoS that a system offers requires the performance of 
three separate tasks, these being: 

• measurement of the system state at all the levels discussed previously; 

• decision and prediction based on observed system state; and 

1 0 • management of system configuration to enhance and/or manage system state. 

The parameters that are measured by the QoS component of this embodiment can be 
divided into layers as described above. The following table shows examples of the 
parameters and associated level to be measured. Note this is not an exhaustive list; 
other embodiments may require monitoring of additional or fewer parameters. Also, 
15 Level 2 parameters depend on the particular application server and database server 
deployed in a particular embodiment. 



19 



Level 


Parameter 


Level 1 

Hardware Infrastructure, Router, 
Processor etc. 


System dependent therefore not defined here. 


Level 2 

Software Infiastructure, Database 
server. Message Broker, Application 
Server, Rule Engine etc. 


Bean Pool Usage, DB cache usage. 

Number of Active Beans, Messages sent by queue per 

Activations, Messages received by 

Number of Bean queue per second, 

Passivations, Bytes send by queue per 

Number of Queued Jobs, second. 

Number of Message Sent Bytes received by queue per 

per second, second, 

DB reads per second. Queue Length, 

DB writes per second. Idle Threads, 

DB cache hits per second. Memory Usage 


Level 3 

Component Performance, Order 
Management System, Risk 
Management System, Message 
Routing System etc. 


Generic Parameters 


Component Specific 


Number of Jobs processed 
per second. 

Time taken to process job. 
Application Status 


Method specific parameters 
such as number of 
invocations per second 


Level 4 

Business Process performance 


Order Round Trip time, System Overall 
Orders Placements per Performance, 
second Number of Concurrent 
Order Cancellations per Users, 
second Number of User Requests 
Order Amendments per per second, 
second. Bandwidth Consumption 
Price Transit time, per User, 
Prices Sent per second per Orders Processed per 
exchange, second per Exchange 
Login time for Users, Order Processed per second 
per User 



Table 1 



All of the parameters described in Table 1 can be measured on a maximum, minimum 
and average basis. The system can also alter the sampling rate that at which these 
5 measurements are taken. For example, the system allows the above parameters to be 
measured over a period ranging from 15 seconds to 5 minutes. It may also log a 
predetermined number (say, ten) best and worst measurements and the time at which 
they occurred. These measurements may be saved to a permanent store for later 
analysis. Figure 5 presents these parameters in diagrammatic form. 

10 Consider the specific example of measurement of timings associated with a message as 
it is handles by the system. 
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A message is time stamped (using UTC to millisecond acciiracy) at the following points 

in the system: as it leaves the user interface (A); when it arrives at the server (B); when 

it leaves the server to the exchange (C); when the exchange responds (D); when it 

leaves the server for transmission to the user interface (E); and when it arrives at the 

5 user interface (F). From these time stamps the following timings can be calculated: 

Total Round Trip Time F-A 

Total Processing Time (C-B) + (E-D) 

Exchange Latency D-C 

Network Latency (B-A) + (F-E) 

Table 2 

Figure 6 shows how the maximum, miniminn and average method invocation times are 
calculated within the individual bean instances and across the bean pool. 

Each bean (A, B and C) within the pool individually coimts the number of invocations 
10 (per relevant method) and the total time taken within each method. They also keep the 
maximum and minimum method invocation times. At the end of the sample period they 
update the respective component manager with the individual counters and reset tibese 
counters for the next period. The component manager then aggregates the individual 
counters to provide pool-based statistics of maximum, minimum and totals. It also 
15 calculates the average transaction time within pool by dividing the 'Total Time Taken 
by Pool' by the 'Total Transaction Processed by Pool' variables (75/7 ~ 10.71ms in the 
example shown in Figure 6). 

The parameters are reported as a snapshot every n seconds, where n is the sampling 
period. The values of die snapshot are based on the aggregated values (as above) of the 
20 individual bean values during this n seconds. The sampling period is configurable on a 
component-by-component basis. 

To implement comprehensive QoS management, measurement of operating parameters 
alone is not sufficient: decisions must be made based upon the parameters measured. 
Likewise, once parameters have been identified and correlated, and a decision or 
25 prediction has been reached, it is advantageous to manage the system actively based on 
these observations to prevent occurrence of problems occurring. This provides a more 
reliable system than one that to reacts to problems as they occur. There will now be 
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described details the QoS mechanisms that can be built into a trading system to 
implemement this. 

Latency and Accuracy of Data Transmission 

This requirement applies to the communication of data fiom the trading system to 
5 external client applications. It is possible to request that the data is sent as fast as 
possible or that data batching may be applied. It is also possible to request whether all 
data changes during the period are to be reported or that only the latest data be reported. 
This communication link support is negotiated during logon to the external application. 
In effect, a client can connect and request that the system batch data (high latency) but 

10 tiiat all changes must be sent, or the client could request that a low-latency link be 
established and that only the latest data is required. This communication link 'quality' 
depends on the requirements of the external applications and the intermediate 
communication link (ISDN, 100Mbit LAN etc.). In response to this request the trading 
system responds by informing the external application whether it can support the 

15 requested link quality or not. It is up to the external application to either renegotiate or 
accept the systems communication quality offer. 

Bandwidth Control 

The system can limit the bandwidth available to a user and ensure that the available 
bandwidth is fairly distributed between clients. Iliere are two aspects to this: firstly to 
20 ensure that bandwidth to which a user has access does not exceed a previously defmed 
limit; and secondly to dynamically limit the bandwidth to which a user has access to 
ensure overall system performance is not degraded. Therefore, the QoS subsystem 
provides a 'fair' allocation of network resources between connected users. 

By this mechanism the QoS subsystem can take remedial action to prevent system 
25 performance from becoming compromised through excessive loading. For example, if 
the QoS subsystem determines that the system as a whole is becoming overloaded, it 
can slow down tiie rate at which users can enter orders until syst^ load has decreased. 
Once load has decrease, it can then once again increase the allowed rate of user input. 
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This is achieved by enabling the system to control bandwidth usage based both on a 
static configuration per user and also dynamically, as will now be described. 

Static Bandwidth Control 

Static bandwidth control is implemented by only allowing a user to submit a fixed 
number x requests per time unit. The time unit is configurable and is also dynamically 
updateable. That is to say, the user does not have to log out and then back m for a 
change in the value x to take effect. 

These request limits are organized around the ability to place, amend, cancel or query 
an order and the total number of request of all types. If a value of zero is specified for 
any of these parameters then the user has unlimited access to the fimction controlled by 
the parameter. An example is set forth in the following tables. 



Parameter 


Value 


Effect 


Place 
Order 


10 


The user can issue up to 10 order placements, or order amendment 
request per second and in total must not exceed 10 requests per 
second. Because the value query and cancel order are zero the 
user has unlimited access to these requests so can issue more than 
10 requests per second in total. 


Amend 
Order 


10 


Query 
Order 


0 


Cancel 
Order 


0 


Total 
Request 


10 


TimeUnit 


1 


Tables 


Parameter 


Value 


Effect 


Place 
Order 


10 


The user can issue up to 10 order placements and up to one order 
amendment request per second, and in total must not exceed 10 
requests per second. Because the value query and cancel order are 
zero the user has unlimited access to these requests so can issue 
more than 10 request per second in total. 


Amend 
Order 


1 


Query 
Order 


0 


Cancel 
Order 


0 


Total 
Request 


10 


TimeUnit 


1 
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Parameter 


Value 


Effect 


Plsce Order 




The user can issue unlimited request to place, amend, query and 
cancel orders. 


Amend 
Order 


0 


Query Order 


0 


Cancel 
Order 


0 


Total 
Request 


0 


TimeUnit 


0 



Parameter 


Value 


Place 


10 


Order 




Amend 


5 


Order 




Query 


5 


Order 




Cancel 


10 


Order 




Total 


10 


Request 




TimeUnit 


5 



Effect 



The user can issue up to 10 order placement or order cancellations, 
and up to five order amendments or order status queries in any 
five-second period. The user may not exceed ten requests in total 
in any five-second period. 



The time period of this requirement is treated as a rolling window and not as an absolute 
5 time period. This requirement is conveniently implemented as a modified 'token 
bucket' (TB) algorithm as detailed below. The general process is illustrated in Figure 7. 

The request-specific tokens (Place, Amend, Query and Cancel) are generated at rate r 
which is TimeUnit / RequestSpecificRate. In other words the system generates four 
token specific rates: 

10 tpiace = TimeUnit / PlaceOrderRate 

ramaui = TimeUnit / AmendOrderRate 

r query = TimeUnit / QueryOrderRate 

r cancel = TimeUnit / CancelOrderRate 
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The tokens are placed into the relevant request-specific bucket. Additionally the 'total 
request' tokens are generated at rate 

r total = TimeUnit / TotalRequest 

and placed into the 'total rate' token bucket. Tokens are placed into the buckets until 
5 the request rate (PlaceOrderRate, AmendOrderRate etc) is met, at which time additional 
tokens are ignored. This is termed the 'depth'. Therefore only a maximum of 'rate' 
tokens may be in a bucket at any point in time. 

Note that by setting TimeUnit to zero this disables token creation and therefore 
completely blocks submission of requests. Request type rates (for example, 
1 0 CancelOrderRate) that are set at zero however are still processed correctly. 

Upon receipt of a request the bandwidth control algorithm first determines if the 
specific request rate (PlaceOrderRate, AmendOrderRate etc) is zero. If it is, the request 
is immediately forwarded. Otherwise, the request is forwarded to the request-specific 
bucket. 

15 If a token for this request is available in the request-specific token bucket a token is 
removed and the request is forwarded to the total requests token bucket. Otherwise, the 
request is denied. 

The total requests token bucket acts in a similar fashion. Upon receipt of a request, an 
attempt is make to remove a token fix)m the bucket regardless of the request type. If a 
20 token can be removed then the request is forwarded, otherwise the request is denied. 

This static choking mechanism is implemented at the extremities of the system: in the 
trading client and in the inbound FIX gateway. 

The mechanism by which processing of batches of requests operates will now be 
described. 

25 Each request within the batch is taken into account and as such consumes one token of 
the relevant type. If the number of tokens is exceeded, all tokens are replaced into the 
bucket (to the maximum depth allowed) and the request is rejected as before. For 
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example, assume that the user can place 10 orders per second and that they submit a 
batch of 15 orders. Fifteen tokens would need to be consumed but only ten are 
available therefore the batch is rejected and the ten consumed tokens are place back into 
the bucket. 

5 The parameters (OrderTokenRate, TimeUnit etc) are defined at a user group level and 
not at the individual user level in this embodiment. All users within a group will 
operate in parallel to each other with respect to the parameter settings. Additionally 
there is a requirement for a 'Disabled User' group to be created. Users in this group 
have the UnitTime set at zero. Users can be placed in this group to stop them entering 
1 0 requests into the system. 

Dynamic bandwidth control 

Dynamic bandwidth control is implemented using a throttling mechanism. The first 
place at which dynamic bandwidtii control occurs is located at the client-side object 
interflEice and the second is implemented in the messaging fa9ade of the infi^tructure 
15 component. Note this throttling is in addition to the message flow control and queue 
length controls of the MOM. 

This throttling must support dynamic reconfiguration through the QoS management 
console and, in the case of user input throttling, through tiiie systems administration 
component during user set-up to define a default bandwidth quota. Equally, a 

20 mechanism to dynamically control user input throttling is provided. 

The message facade bandwidth control will be automatically controlled by the system. 
By this arrangement, as system performance limits are reached, ttie QoS subsystem 
automatically begins to throttle message througihput to restore system performance. 

The Token Bucket Algorithm 

25 As it is central to operation of bandwidth control in this embodiment, operation of the 
token bucket algorithm will now be described with reference to Figure 8. Naturally, all 
of the objects described here are software objects that can be implemented in many 
ways. 
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Tokens are placed in a 'bucket' at a predetermined rate r (for example, five per second). 
Tokens accximulate in the bucket to a maximum depth of D. Tokens placed in the 
bucket once the maximum depth has been reached are discarded. 

When messages arrive (Data In), each message takes a token from the bucket and passes 
5 through (Data Out). If no token can be taken from the bucket the message must wait 
until a token is available, or be discarded. 

The rate of token addition r controls the average flow rate. The depth of the bucket D 
controls the maximum size of burst that can be forwarded through. 

Priority Traffic Routing 

10 In a communications route that is operating within a bandwidth target certain types of 
message must be delivered before others. In this embodiment, message routing priority 
can be altered in dependence upon customer or business requirements. For example, an 
organisation may configure the system such that trades placed by an internal trader have 
a higher delivery priority than trades placed through a 'black box' trading application. 

IS The prioritisation of routing may also be used to ensure that a given SLA is being met 
by dynamically altering message priority to ensure timely processing through the 
system. A user may also have a requirement to prioritise certain traffic types (for 
example order cancellations) over other traffic types (for example order placement). 
This delivery prioritisation is applied at both the message and queue level and can be 

20 altered dynamically through the QoS management console. 

Traffic prioritisation can be divided into the following areas: general message priority 
(MP) and user group priority (UP). 

The MP and UP areas messages are divided into two general categories these being 
normal priority (NP) and expedited priority (BP). There is also a prioritisation category 
25 (PC) on the message type. This provides for all message of a given type to be expedited 
regardless of whether the message was initiated by a trader within the normal priority 
group or expedited group. There is also the concept of queue prioritisation (QP). This 
can be applied to ensure that all messages to Liffe, for example, are processed before 
messages to any other exchange. 
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Therefore, the system can prioritise based upon type of message (MP), user the message 
originated from (UP) and also override the priority if required using the prioritisation 
category (PC). The examples presented in the following tables will make this clearer. 

The example of Table 7 shows how cancellations are always sent before any other 
message within user group, and messages from users within group B are always sent 
before messages form users in groups A. To arrive at the priority, add the relevant 
MP+UP+PC together to a maximum number of 9. 



Message 
Type 


MP 


PC 




Add 


NP 


NP 




Cancel 


EP 


NP 




Amend 


NP 


NP 




Query 


NP 


NP 







NP 


EP 


MP 


1 


2 


UP 


1 


3 


PC 


1 


1 





A 


B 


Add 


1+1+1=3 


1+3+1 =5 


Cancel 


2+1+1 =4 


2+3+1 = 6 


Amend 


1+1+1 =3 


1+3+1 =5 


Query 


1+1+1=3 


1+3+1 =5 



The example of Table 8 shows how query requests are always sent before any other 
message regardless of user, but within user prioritisation, cancellations are sent first and 
B's messages are always sent before A's messages. Also note that queries firom higher 
priority users are given preference over normal priority users. 



Message 
Type 


MP 


PC 




User 


UP 


Add 


NP 


NP 




A 


NP 


Cancel 


EP 


NP 




B 


EP 


Amend 


NP 


NP 






Query 


NP 


EP 









NP 


EP 


GMP 


1 


2 


UMP 


1 


3 


PC 


1 


5 





A 


B 


Add 


1+1+1=3 


1+3+1 =5 


Cancel 


2+1+1=4 


2+3+1 = 6 


Amend 


1+1+1=3 


1+3+1 = 5 


Query 


1+1+5=7 


1+3+5 = 9 



1 5 Dynamic Reconfiguration of System Components 

System components of the embodiment are dynamically configurable to remove the 
need to stop and restart the component. For example, it must be possible to reconfigure 
a component to enable log file production and then at a later stage disable this log file 
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production, without having to stop and start the component. Also these configuration 
parameters are centrally stored to ease configuration management and control. 

The QoS subsystem built into any one embodiment may not provide all of these 
complex measurement and decision support functionalities directly. However, it is 
5 clearly to be preferred that it provides support for them. Moreover, it is preferred that 
the QoS systems are designed in a way to allow integration mto existing management 
facilities that a user may possess. 

JAVA, JMX and JAVABEANS are registered trade marks of Sun Microsystems, Inc. 
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Claims 



1. A trading system comprising a quality-of-service (QoS) subsystem, which 
subsystem is operative to impose limitations upon trading activities in order that 
the performance of a component of the system or of the system as a whole is 
maintained within specified tolerances. 

2. A trading system according to claim 1 in which the QoS subsystem imposes a 
limit upon the rate at which data can enter the system. 

3. A trading system according to claim 2 in which the QoS subsystem limits the 
number of requests that will be accepted on an input. 

4. A trading system according to claim 3 in which the QoS subsystem controls 
input the number of requests that can be made in a time slice. 

5. A trading system according to any one of claims 1 to 3 in which the QoS 
subsystem imposes a limit on the size of burst data that may be received into the 
system in a time slice. 

6. A trading system according to any one of claims 1 to 4 in which the token 
bucket algorithm is used in order to limit the flow of requests into the system. 

7. A trading system according to claim 6 in which the time slice is a sliding time 

slice. 

8. A trading system according to any preceding claim in which the QoS subsystem 
operates such that the system provides a level of service that is dependent upon 
the identity of a user from which the service originates or to whom it is directed. 
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9. A trading system according to any preceding claim in which the QoS subsystem 
operates such that the system provides a level of service (hat is dependent upon 
the nature of a service that is requested. 

10. A trading system according to any preceding claim in which the QoS subsystem 
is operative to measure its performance and dynamically reconfigure operation 
of the system based on these measurements to ensure a defined level of quality- 
of-service. 

1 1. A trading system according to any preceding claim in which the QoS subsystem 
is operative to increase restrictions on users' access to the system as its load 
exceeds a predefined limit. 

12. A trading system according to any preceding claim in which the QoS subsystem 
is operative to assign a priority to a message, messages with a hig^ priority 
being handled in preference to those with a low priority. 

13. A trading system according to claim 12 in which the priority is determined in 
accordance with one or more of the sender of the message, the recipient of the 
message or the content of the message. 

14. A trading system according to claim 12 or claim 13 in which the priority is a 
numerical value that is calculated by addition of contributed values derived fi-om 
one or more of the sender of the message, the recipient of the message or the 
content of the message. 

15. A trading system according to any preceding claim in which the QoS subsystem 
is operative to control latency and accuracy of communication of data fi:om the 
trading system to external client applications. 

16. A trading system according to claim IS in which the client application may 
request that the data is sent as fast as possible or that data batching may be 
applied. 
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17. A trading system according to claim 15 or claim 16 in which the client 
application may request that all data changes during a period are to be reported 
or that only the latest data be reported. 

18. A trading system according to any preceding claim in which the QoS subsystem 
monitors performance of the application by way of Java management 
extensions. 

19. A trading system according to any preceding claim that utilises a rule-based 
system to control alarm reporting, fault diagnosis and reconfiguration. 

20. A trading system substantially as described herein with reference to the 
accompanying drawings. 
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